Improvements and Evaluations of Tactile Graphical Viewer for the Visually Impaired

Our tactile graphical viewer recognizes the type of a handwritten input curve and displays it on a tactile display. We propose new recognition method and change function for input curves. With this recognition method, we propose a procedure to add new curve types. Using the change function, we are able to reduce the time taken to create the required curves. This system has functions to provide some graphical information as a tactile images. We present some evaluations for receiving the information and understanding the underlying properties. After the user inputs and arranges the curves, he/she sends them to a tactile display. The curves can be displayed in a mathematical class with some explanation. For example, the graph of y=x is a straight line sloping from upper right to the lower left. We can easily relate this curve shape to the property “monotone increasing.” To understand this relation, the receiver must recognize the shape “slope upward to the right.” We have to check whether the receiver recognizes the corresponding graphical properties from the output of a tactile display. We prepare some graphical content and questions related to the content, check whether the graphical properties have been understood. We obtain the ratio of correct answers and the average time taken to give an answer and use them to evaluate understanding of the receiver. 2. Outline of the System Our system is a tactile graphical viewer for supporting the visually impaired. A user draws curves using a mouse, and the system recognizes the curve types and corresponding parameters. After arranging the positions or shapes, the curves are send to a tactile display. This section explains the functions of the system. 2.1 Hardware Our system runs on a standard Windows PC with a keyboard and a mouse. We use a tactile display, Dot View (DV-1, KGS Corp.), for obtaining a tactile output. The display has 768 active pins in a 24 × 32 dot display array. 2.2 Objective of the system We consider a situation wherein visually impaired students attend a science class in which their caregivers use our system. During the class, a teacher draws freehand graphs on the blackboard or gestures to express graphical information. After that, the teacher explains some properties using these graphs. The caregivers draw the curve and the students receive these graphical objects using tactile display. If the drawn curves are too complex, they cannot be expressed on the tactile display, the students can not understand even if the contents can be expressed on the display. Then, the caregivers select a few of the curves if necessary. In consideration of this situation, the system must satisfy the following requirements. 1. Facilitate and rapid input of graphical objects 2. Provide simple edit functions to change the position and the shape of a curve. 3. Select function to output to the tactile display. 2.2 Basic functions In this subsection, we explain the standard use of the system. Our target curve types are circles, straight lines, parabola curves, exponential and log curves, cubic curves, sine curves, and tangent curves. The shape of a cubic function with one stationary point differs significantly from that with two stationary points. We consider these curves to be of different types. The case with no stationary point is very rare to deal with; hence, we do not consider this case. Thus, we have nine curve types. We consider a new recognition method for adding new curve types; we will explain the details of this method in the next section. Figure 2.1 Dot View (DV-1) The curve type and curve parameters are recognized. Then, the hand-drawn curve is replaced by a compensated one, and the corresponding parameters are listed in the text area. Click the “Set” button to complete the input of the curve. An elliptical icon appears to the left. We have two methods to change the position and shape of an input curve. While method involves changing the parameters directly in the text area, the other involves dragging a reference point on the selected curve. By left clicking an elliptical icon, we can switch between the modes “display” and “not display,” and the curves with the mode “display” are send to the tactile display. 3. Recognition Method In our method for recognizing the curve types, qualitative measurements are quantified, and a Choquet integral with respect to a non-additive measure is a suitable tool for the analysis. In this section, we improve our recognition method for the systematic addition of recognition targets. 3.1 Recognition method using Choquet integral For the recognition method in the previous version, we prepared nine feature values [5]: 1. Distance from the start point to the end point. 2. Product of the direction angles at the start point and end point (direction angle is the angle between (1,0) and the tangent vector). 3. Difference between the direction angles at the start point and end point. 4. Number of stationary points. 5. Difference between the maximum direction angle and minimum direction angle (direction angles are in [−π ,π] ). 6. Difference between two y -coordinates of the start point and end point. 7. Direction angle at the inflection point. 8. Number of points above the x-axis. 9. Number of points to the right of the y-axis. Figure 2.2 Screen shot of the system. Moreover, for each curve type, we defined a two-additive measure μ . Equivalently, we set real numbers {μk , ν j ,k : j≤9, j<k≤9} , and the corresponding Choquet integral is expressed by ∫ f d μ=∑ j=1 9 f ( j)μ j+∑ j<k f ( j)⊗ f (k )ν j , k , (1) where f is a [0,1] -valued function defined on {1,2,. .. ,9}. Let {yk}k=1 9 be a set of feature values, and set y⃗=( yk , y j⊗ y k)k≤9, j<k and μ⃗=(μk ,ν j ,k )k≤9, jr })dr =( f (a)∨ f (b) – f (a)∧ f (b))α . (2) Thus we consider the operator ∨ and generalize this operator to a general t-conorm: ∫A f d μ⃗=∑x∈A f (x )μ⃗x+∑{ x , y}⊂A f ( x)⊗ f ( y ) ν⃗ x , y (t ) +∑{x , y}⊂A f (x )⊕ f ( y ) ν⃗x , y (c) , (3) where, ⊗ is a t-norm and ⊕ is a t-conorm. In this study, we use Dombi's t-norm, and the standard tconorm defined by using Dombi's t-norm: x⊗ y= 1 ((1 / x−1)+(1/ y−1)) /λ (in this case, λ=2.5 ; see [5] and [6]), Figure 3.1 Example x⊕ y=1−(1−x )⊗(1− y) . We also use the basic unit vectors in 81-dimensional Euclidian space for the set of vectors {μ⃗j , ν⃗ j ,k (t ) , ν⃗ j , k (c) : j≤9, k< j } . 3.3 Adding new curve types Using the vector-valued integral in (3) we change the recognition method as follows: 1. Obtain data for correct curve types. 2. Using feature values and the integral in (3), the curve data are transformed to groups of vectors. 3. For each group, we obtain the mean vector and the covariance matrix. 4. Define the Mahalanobis metrics for each group. 5. The nearest group is the recognition result for a given curve. This is a standard recognition method using feature vectors, and adding new recognition targets is not difficult. We explain the method in deiail. The curve data are created using our system (previous version). We need sufficient data for each curve type (20 to 30 sets of data for each curve type). There must be several data for a standard curve shape, and we also need several data for mistakable shape comparing with other curve type. We obtain an integral vector defined by (3) for each curve. We use standard unit vectors for the definition in (3); however, these are not the optimal choice in general. We will need to obtain more adequate choices through more accurate analysis. After we obtain a vector group for each curve type, we obtain the mean vector and the covariance matrix. The covariance matrix S is a non-negative definite symmetric matrix, and all eigenvalues are non-negative. In general, some feature values may take almost identical values in a certain group. Further, some eigenvalues take very small values and small errors will give large influences to the inverse matrix S−1 . To avoid this situation we consider S+ε I instead of S , where I is a unit matrix and ε>0 is a small number (we use {largest eigenvalue}× 10−8 in practice). (S+ε I )−1 is quite different from a pseudo-inverse. In this analysis, we may use a pseudo-inverse, ignoring eigenvectors corresponding to small eigenvalues. Then, the distance for the difference of these directions is zero with respect to the corresponding Mahalanobis metric. Originally, such differences in these directions are fatal errors when they belong the same group, that is, the distance must be very large. 3.4 Evaluation of recognition method